Repository navigation
test(e2e): add per-feature verification suite and workflow - #255
Merged
Merged
Conversation
Add tests/feature_matrix.yaml with one entry per product feature (call sites, test, lanes), a lane-scoped pytest suite that drives the real clawops/openclaw/acpx install, a deterministic local model server, an ACP stub agent for vendor adapters, and a feature-verification workflow that reports PASS/FAIL/UNVERIFIABLE per feature.
Declare the fake model provider as an explicit models.providers entry after setup, bring sidecars up through clawops when the service manager did not, run hypermemory/context through varlock as the runtime does, redact env-contract secrets from recorded results, and probe the fake server without proxies.
Use OpenClaw's custom-local marker for the fake provider key, prove the OTLP metrics pipeline end to end, read the Qdrant stamp from the stack's own compose state dir, drive smart extraction past its message floor, and split the default Ollama embedding lane into its own feature.
Reference the key through OpenClaw's provider env marker so it is never stored or audited as plaintext, keep fresh-host's embedding default out of lane commands, restore Qdrant when the reset feature fails, and let bootstrap provision Node on lane hosts.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an end-to-end feature verification suite: one assertion-based test per StrongClaw feature, run against the real OpenClaw/ACPX versions that
clawops bootstrapinstalls fromplatform/node-tools.tests/feature_matrix.yaml: 102 features. Each lists its call sites (file:line), its test, its lanes (platforms are derived from the lanes), and a verification level (live,fake-model,stub). 9 features areunverifiablewith a stated reason (live Telegram/WhatsApp, vendor coding-agent sessions, third-party vault backends, the TTY model wizard, the Langfuse stack,devflow run, and two doc-only procedures).tests/suites/e2e/feature_verification/: the suite. It only runs inside a provisioned lane; elsewhere it is deselected, somake testand nestedclawops baseline verifynever touch it..github/workflows/feature-verification.yml: runs onworkflow_dispatchand weekly. Lanes arelinux-core(hypermemory, systemd activation),linux-lancedb,linux-profiles(no host services), andmacos-launchd. A report job joins the lane results with the matrix and writes a per-feature PASS/FAIL/UNVERIFIABLE table to the step summary plus afeature-verification-reportJSON artifact.models.providersentry. Vendor ACP adapters are replaced by an ACP stub agent behind the realacpx. No real keys or secrets are used.use_host_homeputs Linux lanes in the runner HOME so systemd user units can be activated for real.tests/suites/contracts/repo/test_feature_matrix_contract.pychecks that:clawopscommand, profile, backend auth mode and compose stack is covered;Baseline (openclaw 2026.3.13 / acpx 0.3.0)
A branch run recorded 73 PASS, 20 FAIL and 9 UNVERIFIABLE. The FAILs include several pre-existing product bugs, for example:
platform/systemd/openclaw-sidecars.servicerequiresdocker.service, which a user manager cannot resolve, so Linux service activation fails;recovery/backends.py:49passes a positional path toopenclaw backup create, which takes--output;plugins.allowlists disable OpenClaw's bundledollamaprovider;platform_verify.py:235ss parser reads the peer column.The post-merge baseline run and its full evidence are reported separately. This PR fixes none of the product bugs.